<html>

<head>
<title>OPARI: OpenMP Pragma And Region Instrumentor</title>
</head>

<body bgcolor="#FFFFFF">

<div align="center">
<img src="opari-logo-100.gif">
</div>

<h1 align="center"><font color="#FF0000">O</font>penMP
<font color="#FF0000">P</font>ragma
<font color="#FF0000">A</font>nd
<font color="#FF0000">R</font>egion
<font color="#FF0000">I</font>nstrumentor - </h1>
<div align="center">
Version 1.1, October 2001
<p>
&copy; 2001 Forschungszentrum J&uuml;lich, ZAM, Germany
<br>Bernd Mohr
</div>

<p>

<div align="justify">

<font color="#0000FF">OPARI</font> is a source-to-source translation tool
which automatically adds all necessary calls to the
<font color="#0000FF">pomp</font> runtime measurement library which
allows to collect runtime performance data of Fortran, C, or C++ OpenMP
applications. It is based on the idea of OpenMP pragma/directive
rewriting which is described in detail in a paper
(<a href="lacsi01.pdf">PDF</a>) 
for <a href="http://lacsi.lanl.gov/symposium/">LACSI'01</a>.

<p>

<font color="#0000FF">OPARI</font> was developed as part of the
<a href="http://www.fz-juelich.de/zam/kojak/">KOJAK</a> and
<a href="http://www.acl.lanl.gov/tau/">TAU</a> projects.

<h2><hr>DOWNLOAD<hr></h2>

This software is <b>free</B> but <b>copyright</B> &copy; 2001 by
Forschungszentrum Juelich, ZAM, Germany. By downloading and using this
software you automatically agree to comply with the regulations as
described in the <a href="LICENSE">license agreement</a>.

<p>
<table border=1>
<tr>
    <th colspan=3>Sources in gzipped tar format
</tr>
<tr>
    <th>Version
    <th>Date
    <th>Description
</tr>
<tr>
    <td>1.1
    <td>17-Oct-2001
    <td><a href="ChangeLog">Changes</a>
</tr>
<tr>
    <td>1.0
    <td>28-Aug-2001
    <td>Initial version
</tr>
</table>
<p>

<h2><hr>USAGE<hr></h2>

Before compiling the source files of an OpenMP application, each file
needs to be transformed by a call to the <font color="#0000FF">OPARI</font>
tool. In addition, the application has to be linked against the
<font color="#0000FF">pomp</font> runtime measurement library and the
<font color="#0000FF">OPARI</font> runtime table file. The latter has to
be generated by using the <tt><b>-table</b></tt> option to
<font color="#0000FF">OPARI</font> either together with the
transformation of the <b>last</b> input source file or with a separate call
to <font color="#0000FF">OPARI</font> after all transformations are done.
A call to <font color="#0000FF">OPARI</font> has the following syntax:
</div>
<pre><b>
     opari [-f77|-f90|-c|-c++] [-nosrc] [-disable <i>constructs</i>]
           [-rcfile <i>file</i>] [-table <i>tabfile</i>] <i>infile</i> [outfile]
</b>or:<b>
     opari [-rcfile <i>file</i>] -table <i>tabfile</i>
</b></pre>
<div align="justify">

The options and parameters have the following meaning:

<table cellspacing=10>
<tr>
  <td valign="top"><tt><b>[-f77|-f90|-c|-c++]</b></tt>
  <td valign="top">[OPTIONAL]&nbsp;&nbsp;Specifies the programming language
      of the input source file. This option is only
      necessary if the automatic language detection
      based on the input file suffix fails.
</tr>
<tr>
  <td valign="top"><tt><b>[-nosrc]</b></tt>
  <td valign="top">[OPTIONAL]&nbsp;&nbsp;If specified,
      <font color="#0000FF">OPARI</font> does not generate
      <tt><b>#line</b></tt> constructs in the transformation process
      which allow to preserve the original source file
      and line number information. This option might
      be necessary if the OpenMP compiler does not
      understand <tt><b>#line</b></tt> constructs. The default is to
      generate <tt><b>#line</b></tt> constructs.
</tr>
<tr>
  <td valign="top"><tt><b>[-rcfile <i>file</i>]</b></tt>
  <td valign="top">[OPTIONAL]&nbsp;&nbsp;<font color="#0000FF">OPARI</font>
      uses the file <tt><b>./opari.rc</b></tt> to preserve state information
      between calls to <font color="#0000FF">OPARI</font> if the OpenMP
      application consists of more than one source file. With the
      <tt><b>-rcfile</b></tt> option the file <tt><b><i>file</i></b></tt> is
      used instead.  This can be useful if more than one application is
      stored in the same directory or if the source files of an application
      are stored in more than one directory.
</tr>
<tr>
  <td valign="top"><tt><b>-table <i>tabfile</i></b></tt>
  <td valign="top">Generate the <font color="#0000FF">OPARI</font> runtime
      table in file <tt><b><i>tabfile</i></b></tt>.
      This option has to be used either together with the
      call to <font color="#0000FF">OPARI</font> for the transformation of
      the <b>last</b> input source file or with a separate call to
      <font color="#0000FF">OPARI</font> after all transformations are done.
</tr>
<tr>
  <td valign="top"><tt><b>-disable <i>constructs</i></b></tt>
  <td valign="top">[OPTIONAL]&nbsp;&nbsp;Disable the instrumentation of 
      the more fine-grained OpenMP constructs such as <tt><b>!$OMP</b></tt>
      <tt><b>ATOMIC</b></tt>. <tt><b><i>constructs</i></b></tt>
      is a comma separated list of the constructs for which the
      instrumentation should be disabled. Accepted tokens are
      <tt><b>atomic</b></tt>, <tt><b>critical</b></tt>, <tt><b>master</b></tt>,
      <tt><b>flush</b></tt>, <tt><b>single</b></tt>, or <tt><b>locks</b></tt>
      as well as <tt><b>sync</b></tt> to disable all of them.
</tr>
<tr>
  <td valign="top"><tt><b><i>infile</i></b></tt>
  <td valign="top">Input file name.
</tr>
<tr>
  <td valign="top"><tt><b>[<i>outfile</i>]</b></tt>
  <td valign="top">[OPTIONAL]&nbsp;&nbsp;Output file name. If not specified,
      <font color="#0000FF">OPARI</font> uses the name
      <tt><b><i>infile</i>.mod.<i>suffix</i></b></tt> if
      the input file is called <tt><b><i>infile</i>.<i>suffix</i></b></tt>.
</tr>
</table>

In addition to the modified output file, <font color="#0000FF">OPARI</font>
also generates a file named <tt><b><i>infile</i>.opari.inc</b></tt>. It
contains OpenMP region descriptors, one for each OpenMP region found in the
input file. The meaning of the descriptor fields are described in the file
<tt><b>pomp_lib.h</b></tt>.

<p>

In summary, the typical usage of <font color="#0000FF">OPARI</font> consists of the following steps:

<ol>
<li>Reset <font color="#0000FF">OPARI</font> state information by removing
    the state information file if it exists.
    <pre><b>
    % rm -f opari.rc
    </b></pre>

<li>Call <font color="#0000FF">OPARI</font> for each input source file
    <pre><b>
    % opari file1.f90
    ...
    % opari fileN.f90
    </b></pre>

<li>Generate the <font color="#0000FF">OPARI</font> runtime table and
    compile it using a ANSI C compiler
    <pre><b>
    % opari -table opari.tab.c
    % cc -c opari.tab.c
    </b></pre>

<li>Compile all modified output files <tt><b>*.mod.*</b></tt> using the OpenMP
    compiler
    <p>

<li>Link the resulting object files against the
    <font color="#0000FF">OPARI</font> runtime table
    <tt><b>opari.tab.o</b></tt> and the <font color="#0000FF">pomp</font>
    runtime measurement library.
</ol>

<h2><hr>LIMITATIONS<hr></h2>

<font color="#0000FF">OPARI</font> understands all OpenMP constructs by the
<a href="http://www.openmp.org/specs/">Fortran 77/90 OpenMP 2.0</a> and
<a href="http://www.openmp.org/specs/">C/C++ OpenMP 1.0</a> specifications
as well as the OpenMP extension <tt><b>INST</b></tt> directives/pragmas
and the alternative <tt><b>POMP</b></tt> sentinel proposed in
<a href="lacsi01.pdf">lacsi01.pdf</a>.

<h3>Limitations due to fuzzy parsing</h3>

Because <font color="#0000FF">OPARI</font> does not contain full parsers for
the supported programming languages the following restrictions apply:

<h4>Fortran 77/90:</h4>

<ol>
<li> The <tt><b>!$OMP</b></tt> <tt><b>END</b></tt> <tt><b>DO</b></tt> and
     <tt><b>!$OMP</b></tt> <tt><b>END</b></tt> <tt><b>PARALLEL</b></tt>
     <tt><b>DO</b></tt> directives
     are required (and not optional as described in the OpenMP specification)
<p>
<li> The <i>atomic expression</i> controlled by a
     <tt><b>!$OMP</b></tt> <tt><b>ATOMIC</b></tt>
     directive has to be on a line all by itself.
<p>
<li> If the measurement environment does not support the automatic
     recording of user function entries and exits, the
     <font color="#0000FF">OPARI</font> runtime
     measurement library has to be initialized by a
     <tt><b>!$OMP</b></tt> <tt><b>INST</b></tt> <tt><b>INIT</b></tt>
     directive prior to any other OpenMP directive.
</ol>
     
<h4>C/C++:</h4>

<ol>
<li> <i>structured blocks</i> describing the extend of an OpenMP pragma
     need to be either compound statements <tt><b>{....}</b></tt>,
     <tt><b>while</b></tt> loops, or simple statements. In addition,
     <tt><b>for</b></tt> loops are supported after
     <tt><b>omp</b></tt> <tt><b>for</b></tt> and <tt><b>omp</b></tt>
     <tt><b>parallel</b></tt> <tt><b>for</b></tt> pragmas.
     Complex statements like
     <tt><b>if-then-else</b></tt> or <tt><b>do-while</b></tt> need to
     be enclosed in a block ( <tt><b>{....}</b></tt> ). 
<p>
<li> If the measurement environment does not support the automatic
     recording of user function entries and exits, the
     <font color="#0000FF">OPARI</font> runtime measurement library has to
     be initialized by a <tt><b>omp</b></tt> <tt><b>inst</b></tt>
     <tt><b>init</b></tt> pragma prior to any other OpenMP pragma.
</ol>
   
We did not find these limitations overly restrictive during our tests and
experiments. They rarely apply for well-written code. If they do, the
original source code can easily be fixed.  Of course, it would be possible
to remove these limitations by enhancing <font color="#0000FF">OPARI</font>`s
parsing sophistication.

<h3>Limitations due to source-to-source translation</h3>

In addition, because of some subtleties in the OpenMP standard specifications,
the transformations performed by <font color="#0000FF">OPARI</font> on the
source code level can differ from the same instrumentation done by a real
OpenMP compiler. Here is the list of limitations we currently know about:
<ul>
<li><font color="#0000FF">OPARI</font> makes implicit barriers explicit.
    Unfortunately, this method cannot be used for measuring the barrier waiting
    time at the end of <tt><b>PARALLEL</tt></b> directives because they do not
    allow a <tt><b>NOWAIT</tt></b> clause. Therefore, we add an explicit
    barrier with corresponding performance interface calls here. For <font
    color="#0000FF">OPARI</font>, this means that actually two barriers get
    called. But the second (implicit) barrier should execute and succeed
    immediately because the threads of the OpenMP team are already synchronized
    by the first barrier.
<p>
<li>The OpenMP standard (unfortunately) allows compilers to ignore
    <tt><b>NOWAIT</tt></b>s, which means that in this case <font
    color="#0000FF">OPARI</font> inserts an extra barrier and the
    <font color="#0000FF">pomp</font> functions get invoked on this extra
    (and not the real) barrier.
<p>
<li><font color="#0000FF">OPARI</font> cannot instrument the (required)
    internal synchronization inside <tt><b>!$OMP</tt></b>
    <tt><b>WORKSHARE</tt></b>.
<p>
<li>Mark Bull's <a
    href="http://www.epcc.ed.ac.uk/research/openmpbench/">Microbenchmarks</a>
    show that some compiler use different implementations (with different
    characteristics) for implicit and explicit barriers. If <font
    color="#0000FF">OPARI</font> changes implicit to explicit barriers, we
    measure the wrong behavior on these compilers.
</ul>

Of course, an OpenMP compiler can insert the <font color="#0000FF">pomp</font>
performance interface calls directly around the implicit barriers, thereby
avoiding the described overheads and discrepancies.

</div>
</body>
</html>
